Embodiments of the present invention relate to an image or imager distortion processing method and apparatus to transform an image, optimizing it for a distortion processing algorithm (e.g., dewarping algorithm). Using the same method, embodiments of the invention allow magnification of some areas of interest in the image or imager field of view, optimizing the storage, transmission, resolution, or display of the content.
A two-dimensional (2D) image is created by an imager, such as a narrow angle imager, panoramic imager, wide-angle optical system, fisheye optical system, panomorph optical system, or even image stitching from multiple narrow-angle images or image stitching from multiples panoramic images. No matter what kind of imager is used, the image often contains optical distortion. This behavior is caused by remapping of the three-dimensional (3D) environment to a 2D plane. As the result, contents in the image are deformed, and the exact distortion profile depends on the imager parameters. Usually, the distortion is much more obvious with panoramic imagers.
To properly display an image without distortion, or feed the correct image to any image processing algorithm, the distortion in panoramic image has to be processed, as in U.S. Pat. No. RE 44,087 E for linear distortion or U.S. Pat. Nos. 6,844,990 B2 and 6,865,028 B2 for custom symmetric and custom asymmetric distortion, respectively. These existing distortion processing algorithms or dewarping algorithms (actions to correct, remove, process or modify the distortion of an image) are implemented by software or hardware on different platforms, such as a computer, smartphone, tablet, System on Chip (SoC), or the like. As an example, IPIX plug-in for Milestone system is a dewarping software to correct distortion of fisheye lenses. Some dewarping algorithms use only a linear distribution function to correct different distortion profiles from different panoramic imagers. In this case, because the linear correction function used does not match the real distortion profile of the panoramic image or imager, some distortion remains in the images after dewarping. To properly correct the distortion, the dewarping algorithm has to use the distribution function (distortion profile), considering the distortion profile of the panoramic imager and the application user experience. This means that the departure between the distortion profile and the dewarping algorithm distribution function has to be controlled. One approach could be to input the distortion profile as a mathematical function to the distortion processing algorithm to correct a panoramic image. However this approach cannot be used in all situations, for example, but not limited to, sometimes the distortion profile is confidential or not present in the image processing unit running the distortion processing algorithm. Furthermore, the distortion profile for a new imager cannot be easily added or updated in existing software or in a hardware processing algorithm. This requires a software update, firmware update, or hardware modification (ASIC modification, for example).
ImmerVision Enables 2.0 algorithms correct the distortion of panomorph images or imagers. The libraries including the algorithms contain some distortion profiles (or distribution functions) corresponding to the panomorph imagers or panomorph lenses available at the time of the algorithm release in the image processing units (library, ASIC, FPGA, or the like). If the distortion profile of a panoramic image or imager is in this list of pre-recorded distortion profiles, a user only needs to input a distortion profile reference number and the processing algorithm will use the proper existing distortion profile. For example, at least some embodiments of U.S. Patent Application US 2015/0281507 A1 use a marker on the image to record this distortion profile reference number or the distortion profile itself. In the embodiments where only the reference number of the distortion profile is marked, the distortion processing unit can read the reference number from the marker and automatically use the corresponding distortion profile. Compared with transferring the whole distortion profile to the distortion processing algorithm, this way is more efficient. In some cases, this approach cannot be used since the distortion profile is not present in the algorithm, and therefore the algorithm is not designed to support this new type of distortion profile. An example of this case is a panoramic image with a new distortion profile which is created by, for example, but not limited to, a new wide-angle, fisheye, or panomorph lens, or new image processing algorithm. For this image, the distortion processing unit has to be updated with the new distortion profile or a completely new algorithm before the distortion processing unit can correct, remove, process or modify the distortion of the panoramic image or imager. In some situations, algorithm updates cannot be done easily, especially when the distortion processing algorithm is implemented on hardware. This limits the application of distortion processing algorithms for different panoramic images or imagers.
Due to the big data size of images or videos, the optimization of image storage or transmission resource is always preferred. An optimization approach is to reduce image or video size. For panoramic image or wide-angle images, U.S. Pat. No. 8,238,695 proposes to crop a part of image data for a virtual camera from the whole image to reduce image data size. This solution sacrifices the image data outside of the virtual camera area, and it can only be used for the use case of a virtual camera. The saved or transmitted image only contains image data for the virtual camera, and other image data from the rest of the field of view is lost. This does not work for the application of whole panoramic images. Another alternative, to keep the whole panoramic field of view, is to instead uniformly scale down the whole image to a lower resolution and to decrease the image size. However, this scaling is the same for the whole image, and the image quality is degraded in the whole image. One optimized approach would be to use higher resolution in important image areas and lower resolution in other image areas, meaning different scaling ratios are applied for different areas of the image. In this way, more pixels would be used to emphasize important areas when the image size is decreased. This non-linear scaling creates a new distortion in the generated image. This new distortion is different from the original distortion of the original panoramic image. So the distortion in the original panoramic image cannot be used for processing this optimized image.
U.S. Pat. No. 6,865,028 proposes to project non-disk shape panoramic image on a rectangular image sensor. This includes, but is not limited to, elliptical images. Compared with a circular image, the elliptical image occupies more pixels on the rectangular image sensor and there are therefore fewer black pixels in the output image. In this way, out of the total file size for the image, a larger fraction is used by useful image contents compared to useless black content. The resulting image has higher effective resolution. For image storage and transmission, this elliptical panoramic image containing fewer black pixels is more efficient than a circular panoramic image. However, U.S. Pat. No. 6,865,028 only suggests optical means to implement this method of creating panomorph optical system, and the distortion profile is unique for each panomorph optical system. In addition, the distortion profile is fixed after the panomorph optical system is produced. The distortion profile can only be modified by designing new lenses, so an existing optical lens cannot be modified and cannot dynamically change the distortion depending on applications.
Dynamically changing the distortion has been proposed in the past in U.S. Patent Publication No. 2012/0314065 A1. However, this invention changes the distortion in real-time using a hardware optical system consisting of a deformable optical surface as a deformable mirror. This construction is complex and cannot be used in existing panoramic imagers to dynamically modify the areas of interest.
Since 2006, the Panomorph Geometrical Model Simulator (PGMS) software, developed by ImmerVision, simulates distortions in images to analyze image distortion effects. The Panomorph Geometrical Model was described in the paper “Novel hemispheric image formation: concepts and applications” published in “Proc. SPIE 6994, Photon Management III, 699406 (Apr. 25, 2008)”. The software can create different distortions on images, including different pre-set distortions, non-linear scaling distortions, or any customized distortions. However, the software is not for panoramic image distortion processing, and it cannot transform a panoramic image with a new distortion profile to be optimized for distortion processing algorithms. It cannot transfer panoramic image distortion to a non-linear scaling distortion to emphasize important areas in an image either. Finally, it cannot be integrated into a panoramic image processing pipeline.